Skip to content

fix(prose-tests): prescribe the cross-check, drop a check that cannot fire, lint the scope - #573

Merged
leeovery merged 1 commit into
mainfrom
fix/prose-prescribe-what-we-observed
Jul 27, 2026
Merged

fix(prose-tests): prescribe the cross-check, drop a check that cannot fire, lint the scope#573
leeovery merged 1 commit into
mainfrom
fix/prose-prescribe-what-we-observed

Conversation

@leeovery

@leeovery leeovery commented Jul 27, 2026

Copy link
Copy Markdown
Owner

Three things the full run turned up, none of them failures.

The asserter's cross-check is now prescribed

It had begun comparing the action record against the walker's narrative unprompted, and reporting divergences. Right instinct — but nobody's instruction, so any run depending on it was depending on a coincidence.

Now required, both directions named:

  • Narrated but not recorded — the walker described something it didn't do. The step resting on that claim fails, and the discrepancy is a finding in itself.
  • Recorded but not narrated — usually orientation, fails nothing, but reported: a call that touched state and went unmentioned is the shape a silent repair takes.

The two streams exist so neither is taken on trust. That only holds if they're actually set against each other every time.

engine_before_write declared only where it can fire

It was on all ten cases and could only ever trip on the three that write state. Reporting N/A seven times is honest, but it's still a check dressed as coverage.

The check itself stays — it's the anti-fabrication guard for anything landing state, and state-landing cases are the direction we're heading.

New: the scope lint

A case's files[] isn't documentation — it decides whether editing a file selects that case for a run. Prose the walk goes through but the case omits can be changed without the test covering it ever running again. Every case has such gaps today.

It can't be built by following links: the reachable set is far wider than the walked one, and a case declaring everything reachable would be selected for branches it deliberately stops before (session-loop.md, task-loop.md, conclude-discovery.md…). So it's computed from the walk's own Read calls, and reported as a finding about the case — correcting nothing, since which files belong is an authoring judgement.

Test plan

  • 86/86 across the prose suites, including five new cases for the lint: it names undeclared prose, stays quiet when the list is complete, ignores world-state reads and shell cats, and reports each file once
  • Corpus valid

🤖 Generated with Claude Code

Stack

  1. docs(design): prose-tests programme design log #544
  2. feat(prose-tests): the framework — cases, worlds, runner, skill #545
  3. test(prose): feature happy-path corpus — five worlds, seven cases #546
  4. test(prose): bugfix corpus — the investigation-centric surfaces #548
  5. test: retry recursive teardown removals — kill a class of phantom failures #549
  6. fix(entry-skills): close the handoff fences — six files render their arms wrong #550
  7. docs: a contributing page for working on the system #551
  8. fix(entry-skills): every handoff arm says to invoke the skill #552
  9. fix(implementation): environment setup belongs to the setup reference alone #553
  10. fix(prose-tests): the asserter is told which substitutions were armed #554
  11. feat(prose-tests): the mid-flow substitution, and a world only prose can describe #555
  12. test(prose): claims assert consequences, not what was displayed #556
  13. feat(prose-tests): record everything the agents do, results included #557
  14. fix(discussion-entry): the handoff reports the source it actually had #558
  15. fix(prose-tests): the stop hook records, and names the model that walked #559
  16. fix(prose-tests): command output was never actually recorded #560
  17. feat(prose-tests): judge the walk as told, not the summary returned #561
  18. feat(prose-tests): decide in code what an agent should not be deciding #562
  19. test(prose): a case starts where a session starts #563
  20. feat(prose-tests): walk on Sonnet, judge on Opus, escalate a failure #564
  21. test(prose): give the eight read-only cases something that can fail #565
  22. test(prose): only walks that can be observed, and checks that survive the trip #566
  23. fix(prose-tests): the verdict names only the model the record names #567
  24. test(prose): discovery, walked to the point where work first exists #568
  25. fix(prose-tests): the asserter judges which of prose or walker was at fault #569
  26. docs(conventions): a step whose reference routes every exit still signposts #570
  27. test(prose): discovery's epic arm, to the same durability boundary #571
  28. fix(prose-tests): keep the walk's closing turn, and name the one live difference #572
  29. fix(prose-tests): prescribe the cross-check, drop a check that cannot fire, lint the scope #573 👈 current
  30. test(prose): declare the prose each case actually walks #574
  31. fix(investigation): the symptom interview never ran for work shaped in discovery #575
  32. feat(prose-tests): conduct — playing a person where a script cannot reach #576
  33. fix(prose-tests): what a write put in a file is evidence, not incident #577
  34. fix(investigation): the interview starts from the carrier, not over it #578
  35. test(prose): quick-fix scoping, the whole definition stage in one walk #579
  36. fix(prose-tests): the world log is the walker's record #580
  37. fix(prose-tests): a confirmed failure reports both runs' checks #581
  38. fix(prose-tests): each prescribed command runs as written #582
  39. fix(scoping): context gathering reads the whole carrier, asks only the gaps #583

This was referenced Jul 27, 2026
@leeovery
leeovery force-pushed the fix/prose-prescribe-what-we-observed branch from 3cb2b58 to e7ed77e Compare July 27, 2026 15:49
@leeovery
leeovery force-pushed the fix/prose-closing-turn-and-known-difference branch from bb9b308 to d00a14a Compare July 27, 2026 15:50
@leeovery
leeovery force-pushed the fix/prose-prescribe-what-we-observed branch from e7ed77e to 0344a29 Compare July 27, 2026 15:50
@leeovery
leeovery force-pushed the fix/prose-closing-turn-and-known-difference branch from d00a14a to c15985f Compare July 27, 2026 15:52
@leeovery
leeovery force-pushed the fix/prose-prescribe-what-we-observed branch from 0344a29 to 15960f6 Compare July 27, 2026 15:52
@leeovery
leeovery force-pushed the fix/prose-closing-turn-and-known-difference branch from c15985f to 5c8eaa3 Compare July 27, 2026 15:53
@leeovery
leeovery force-pushed the fix/prose-prescribe-what-we-observed branch from 15960f6 to 3796225 Compare July 27, 2026 15:53
@leeovery
leeovery force-pushed the fix/prose-closing-turn-and-known-difference branch from 5c8eaa3 to b25026e Compare July 27, 2026 15:54
@leeovery
leeovery force-pushed the fix/prose-prescribe-what-we-observed branch from 3796225 to 93eeae7 Compare July 27, 2026 15:54
@leeovery
leeovery force-pushed the fix/prose-closing-turn-and-known-difference branch from b25026e to 72d1fbf Compare July 27, 2026 15:56
@leeovery
leeovery force-pushed the fix/prose-prescribe-what-we-observed branch from 93eeae7 to 8c7ae09 Compare July 27, 2026 15:56
@leeovery
leeovery force-pushed the fix/prose-closing-turn-and-known-difference branch from 72d1fbf to 7f84584 Compare July 27, 2026 15:57
@leeovery
leeovery force-pushed the fix/prose-prescribe-what-we-observed branch from 8c7ae09 to 4073006 Compare July 27, 2026 15:57
@leeovery
leeovery force-pushed the fix/prose-closing-turn-and-known-difference branch from 7f84584 to d5175ed Compare July 27, 2026 15:58
@leeovery
leeovery force-pushed the fix/prose-prescribe-what-we-observed branch from 4073006 to c959550 Compare July 27, 2026 15:58
@leeovery
leeovery force-pushed the fix/prose-closing-turn-and-known-difference branch from d5175ed to c992f1d Compare July 27, 2026 15:59
@leeovery
leeovery force-pushed the fix/prose-prescribe-what-we-observed branch from c959550 to 249f415 Compare July 27, 2026 15:59
@leeovery
leeovery force-pushed the fix/prose-closing-turn-and-known-difference branch from c992f1d to c6d27ea Compare July 27, 2026 16:00
@leeovery
leeovery force-pushed the fix/prose-prescribe-what-we-observed branch from 249f415 to 1a81cbb Compare July 27, 2026 16:00
@leeovery
leeovery force-pushed the fix/prose-closing-turn-and-known-difference branch from c6d27ea to ddd8f1b Compare July 27, 2026 16:01
@leeovery
leeovery force-pushed the fix/prose-prescribe-what-we-observed branch from 1a81cbb to 77b5189 Compare July 27, 2026 16:01
@leeovery
leeovery force-pushed the fix/prose-closing-turn-and-known-difference branch from ddd8f1b to 07bdc40 Compare July 27, 2026 16:02
@leeovery
leeovery force-pushed the fix/prose-prescribe-what-we-observed branch from 77b5189 to 992639b Compare July 27, 2026 16:02
@leeovery
leeovery force-pushed the fix/prose-closing-turn-and-known-difference branch from 07bdc40 to 9fdf6ab Compare July 27, 2026 17:05
@leeovery
leeovery force-pushed the fix/prose-prescribe-what-we-observed branch from 992639b to f6f4e39 Compare July 27, 2026 17:05
@leeovery
leeovery changed the base branch from fix/prose-closing-turn-and-known-difference to main July 27, 2026 17:05
… fire, lint the scope

Three things the full run turned up, none of them failures.

The asserter had begun comparing the action record against the walker's
narrative unprompted, and reporting where they diverged. It is the right
instinct and it was nobody's instruction, so a run that depended on it
was depending on a coincidence. It is now required, with both directions
named: narrated but not recorded means the walker described something it
did not do, and the step resting on that claim fails; recorded but not
narrated is usually orientation and fails nothing, but is reported,
because a call that touched state and went unmentioned is the shape a
silent repair takes. The two streams exist so neither is taken on trust,
which only holds if they are actually set against each other.

engine_before_write was declared on every case and could only ever fire
on the three that write state. Reporting N/A seven times is honest but it
is still a check dressed as coverage, so it is declared only where a walk
can trip it. The check itself stays: it is the anti-fabrication guard for
anything that lands state, and cases that land state are the direction we
are heading.

The scope lint is new. A case's file list is not documentation — it
decides whether editing a file selects that case for a run, so prose the
walk goes through but the case omits can be changed without the test that
covers it ever running again. Every case has such gaps. The list cannot
be built by following links: the reachable set is far wider than the
walked one, and a case that declared everything reachable would be
selected for branches it deliberately stops before. So it is computed
from the walk's own Reads and reported as a finding about the case,
correcting nothing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@leeovery
leeovery force-pushed the fix/prose-prescribe-what-we-observed branch from f6f4e39 to 7fbbbd8 Compare July 27, 2026 17:06
@leeovery
leeovery merged commit 84f4488 into main Jul 27, 2026
@leeovery
leeovery deleted the fix/prose-prescribe-what-we-observed branch July 27, 2026 17:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant